Skip to content

[feat](nereids) support runtime filter for except/intersect (FE part)#49947

Merged
englefly merged 12 commits into
apache:masterfrom
englefly:set-runtime-filter-fe-0410
Jun 25, 2025
Merged

[feat](nereids) support runtime filter for except/intersect (FE part)#49947
englefly merged 12 commits into
apache:masterfrom
englefly:set-runtime-filter-fe-0410

Conversation

@englefly

@englefly englefly commented Apr 10, 2025

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented Apr 10, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@englefly

englefly commented Apr 10, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

doris-robot commented Apr 10, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 34107 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 697891e3de6b79aacc0feb7717bb651fc223a69e, data reload: false

------ Round 1 ----------------------------------
q1	26170	5083	4980	4980
q2	2064	276	189	189
q3	10403	1212	687	687
q4	10235	1013	523	523
q5	7538	2327	2340	2327
q6	190	168	134	134
q7	930	739	617	617
q8	9304	1270	1102	1102
q9	6745	5205	5088	5088
q10	6885	2308	1871	1871
q11	488	308	276	276
q12	353	354	225	225
q13	17783	3693	3094	3094
q14	251	229	210	210
q15	531	484	480	480
q16	648	633	567	567
q17	620	857	391	391
q18	7753	7101	7103	7101
q19	1523	950	576	576
q20	339	338	234	234
q21	4011	3477	2482	2482
q22	1055	1016	953	953
Total cold run time: 115819 ms
Total hot run time: 34107 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5150	5122	5104	5104
q2	242	328	232	232
q3	2157	2644	2283	2283
q4	1451	1795	1406	1406
q5	4531	4494	4367	4367
q6	217	167	129	129
q7	2008	1930	1796	1796
q8	2579	2560	2559	2559
q9	7098	7111	7148	7111
q10	2975	3204	2785	2785
q11	574	492	495	492
q12	690	772	625	625
q13	3565	3859	3294	3294
q14	280	286	269	269
q15	517	470	469	469
q16	638	681	634	634
q17	1122	1580	1334	1334
q18	7734	7572	7274	7274
q19	843	832	985	832
q20	1947	1931	1788	1788
q21	5074	4697	4676	4676
q22	1051	1013	995	995
Total cold run time: 52443 ms
Total hot run time: 50454 ms

@doris-robot

doris-robot commented Apr 10, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 185973 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 697891e3de6b79aacc0feb7717bb651fc223a69e, data reload: false

query1	992	498	466	466
query2	6564	1968	1967	1967
query3	6764	224	222	222
query4	26189	23376	23512	23376
query5	4309	631	491	491
query6	312	208	199	199
query7	4618	503	281	281
query8	303	259	252	252
query9	8624	2593	2594	2593
query10	466	338	264	264
query11	15587	15350	14779	14779
query12	184	110	110	110
query13	1658	521	386	386
query14	9921	5820	5896	5820
query15	207	184	185	184
query16	7637	608	466	466
query17	1175	732	543	543
query18	2021	401	292	292
query19	185	184	151	151
query20	118	117	117	117
query21	209	120	109	109
query22	4081	4152	4033	4033
query23	33794	32687	33062	32687
query24	8352	2377	2379	2377
query25	521	442	382	382
query26	1232	272	147	147
query27	3031	499	330	330
query28	4350	2432	2410	2410
query29	711	561	425	425
query30	285	227	198	198
query31	973	845	769	769
query32	74	65	68	65
query33	555	368	311	311
query34	789	860	526	526
query35	802	839	743	743
query36	949	982	871	871
query37	120	99	79	79
query38	4137	4242	4086	4086
query39	1492	1411	1400	1400
query40	213	118	107	107
query41	57	54	58	54
query42	118	111	106	106
query43	503	519	475	475
query44	1321	817	809	809
query45	178	172	169	169
query46	833	1016	619	619
query47	1770	1795	1717	1717
query48	376	426	296	296
query49	794	529	429	429
query50	642	671	405	405
query51	4121	4160	4068	4068
query52	105	107	106	106
query53	235	263	184	184
query54	580	572	493	493
query55	87	82	82	82
query56	352	310	288	288
query57	1114	1156	1066	1066
query58	266	259	275	259
query59	2647	2769	2613	2613
query60	329	322	312	312
query61	130	128	126	126
query62	782	745	658	658
query63	231	190	191	190
query64	4272	1018	692	692
query65	4351	4242	4271	4242
query66	1100	437	363	363
query67	15843	15444	15389	15389
query68	7849	888	532	532
query69	460	302	270	270
query70	1184	1166	1073	1073
query71	409	333	291	291
query72	5513	4737	4847	4737
query73	655	616	351	351
query74	9040	9171	8877	8877
query75	3260	3207	2701	2701
query76	3366	1182	742	742
query77	533	385	301	301
query78	9964	10064	9287	9287
query79	2249	837	559	559
query80	568	517	437	437
query81	508	264	227	227
query82	462	135	101	101
query83	256	255	249	249
query84	250	112	80	80
query85	776	351	306	306
query86	373	300	285	285
query87	4523	4551	4382	4382
query88	3705	2250	2229	2229
query89	385	310	284	284
query90	1940	216	229	216
query91	192	144	118	118
query92	79	61	63	61
query93	1808	938	582	582
query94	699	398	304	304
query95	373	290	290	290
query96	484	570	279	279
query97	3167	3233	3149	3149
query98	230	204	200	200
query99	1326	1443	1249	1249
Total cold run time: 274227 ms
Total hot run time: 185973 ms

@doris-robot

doris-robot commented Apr 10, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.44 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 697891e3de6b79aacc0feb7717bb651fc223a69e, data reload: false

query1	0.03	0.04	0.03
query2	0.12	0.10	0.10
query3	0.25	0.19	0.19
query4	1.60	0.19	0.20
query5	0.59	0.59	0.60
query6	1.16	0.73	0.72
query7	0.02	0.03	0.02
query8	0.04	0.03	0.04
query9	0.58	0.51	0.50
query10	0.59	0.58	0.58
query11	0.17	0.10	0.10
query12	0.14	0.12	0.12
query13	0.62	0.60	0.62
query14	2.68	2.84	2.70
query15	0.95	0.87	0.87
query16	0.40	0.39	0.39
query17	1.07	1.02	1.02
query18	0.22	0.20	0.20
query19	1.92	1.99	1.90
query20	0.01	0.01	0.01
query21	15.36	0.93	0.54
query22	0.74	1.10	0.66
query23	15.08	1.42	0.58
query24	6.84	0.88	1.09
query25	0.48	0.18	0.10
query26	0.56	0.17	0.13
query27	0.05	0.05	0.05
query28	9.70	0.91	0.44
query29	12.56	3.96	3.31
query30	0.25	0.08	0.06
query31	2.83	0.61	0.39
query32	3.23	0.58	0.48
query33	3.13	3.14	3.12
query34	15.61	5.26	4.52
query35	4.54	4.55	4.50
query36	0.66	0.50	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.02	0.02	0.02
query40	0.18	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.28 s
Total hot run time: 31.44 s

@englefly

englefly commented Apr 11, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

doris-robot commented Apr 11, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 34140 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 57a0bf0c6c50c27e5fe650ae69e95a68e8841b3c, data reload: false

------ Round 1 ----------------------------------
q1	26063	5026	5021	5021
q2	2068	282	184	184
q3	10384	1214	678	678
q4	10230	1011	544	544
q5	7551	2383	2303	2303
q6	191	165	139	139
q7	916	738	606	606
q8	9313	1298	1135	1135
q9	6843	5172	5078	5078
q10	6853	2267	1879	1879
q11	498	293	267	267
q12	356	359	224	224
q13	17785	3659	3130	3130
q14	223	217	213	213
q15	534	487	485	485
q16	623	621	588	588
q17	598	843	382	382
q18	7456	7119	7075	7075
q19	1915	954	554	554
q20	341	336	233	233
q21	4354	3436	2429	2429
q22	1028	993	1002	993
Total cold run time: 116123 ms
Total hot run time: 34140 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5237	5109	5206	5109
q2	233	324	233	233
q3	2175	2639	2257	2257
q4	1431	1890	1514	1514
q5	4575	4418	4465	4418
q6	217	167	133	133
q7	1986	1967	1821	1821
q8	2615	2673	2530	2530
q9	7413	7321	7341	7321
q10	3041	3224	2783	2783
q11	582	517	497	497
q12	711	824	620	620
q13	3520	3909	3373	3373
q14	269	299	267	267
q15	538	497	478	478
q16	643	721	640	640
q17	1167	1528	1396	1396
q18	7831	7439	7419	7419
q19	869	791	912	791
q20	1879	1975	1819	1819
q21	5431	4941	4758	4758
q22	1094	1063	1004	1004
Total cold run time: 53457 ms
Total hot run time: 51181 ms

@doris-robot

doris-robot commented Apr 11, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 169302 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 57a0bf0c6c50c27e5fe650ae69e95a68e8841b3c, data reload: false

============================================
query1	1397	1087	1040	1040
query2	6304	1994	1890	1890
query3	11000	4512	4671	4512
query4	query5	6144	527	467	467
query6	438	199	212	199
query7	5460	285	280	280
query8	332	247	254	247
query9	9417	2615	2587	2587
query10	520	305	264	264
query11	15595	15062	14736	14736
query12	148	108	104	104
query13	1422	532	369	369
query14	10120	5923	5969	5923
query15	199	185	173	173
query16	7181	632	473	473
query17	1059	719	589	589
query18	1616	414	315	315
query19	314	193	167	167
query20	127	120	118	118
query21	217	124	106	106
query22	4006	4142	3973	3973
query23	34135	33676	33651	33651
query24	6825	2502	2457	2457
query25	542	505	469	469
query26	708	290	160	160
query27	2486	519	343	343
query28	3289	2458	2441	2441
query29	662	638	426	426
query30	272	232	205	205
query31	875	875	784	784
query32	83	61	67	61
query33	444	374	303	303
query34	748	876	516	516
query35	796	816	759	759
query36	970	1007	906	906
query37	126	106	81	81
query38	4272	4191	4283	4191
query39	1546	1467	1466	1466
query40	215	119	106	106
query41	59	52	52	52
query42	121	112	109	109
query43	509	510	547	510
query44	1356	846	837	837
query45	186	178	176	176
query46	839	1045	653	653
query47	1872	1881	1800	1800
query48	383	407	319	319
query49	719	504	429	429
query50	638	723	421	421
query51	4228	4232	4185	4185
query52	114	101	95	95
query53	217	260	181	181
query54	592	593	521	521
query55	84	82	85	82
query56	288	293	295	293
query57	1174	1197	1129	1129
query58	269	276	265	265
query59	2725	2828	2738	2738
query60	330	312	311	311
query61	131	131	128	128
query62	746	745	672	672
query63	224	187	188	187
query64	2025	1120	713	713
query65	4441	4366	4304	4304
query66	750	406	316	316
query67	16462	16432	15474	15474
query68	7897	871	509	509
query69	555	295	262	262
query70	1183	1139	1111	1111
query71	487	302	291	291
query72	5325	4693	4597	4597
query73	1271	646	347	347
query74	8922	8929	9050	8929
query75	4139	3199	2674	2674
query76	5644	1183	766	766
query77	760	383	288	288
query78	9764	10091	9215	9215
query79	7259	794	552	552
query80	618	499	423	423
query81	467	256	228	228
query82	423	128	97	97
query83	257	261	240	240
query84	308	106	84	84
query85	754	389	317	317
query86	334	314	272	272
query87	4442	4517	4415	4415
query88	2800	2228	2215	2215
query89	427	300	274	274
query90	1970	214	214	214
query91	147	152	113	113
query92	76	61	57	57
query93	4060	914	561	561
query94	723	409	326	326
query95	375	295	281	281
query96	480	555	275	275
query97	3134	3255	3130	3130
query98	223	222	202	202
query99	1362	1399	1280	1280
Total cold run time: 260078 ms
Total hot run time: 169302 ms

@doris-robot

doris-robot commented Apr 11, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 57a0bf0c6c50c27e5fe650ae69e95a68e8841b3c, data reload: false

query1	0.04	0.03	0.03
query2	0.13	0.11	0.11
query3	0.25	0.20	0.19
query4	1.60	0.20	0.19
query5	0.61	0.57	0.57
query6	1.17	0.70	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.52	0.53
query10	0.56	0.60	0.56
query11	0.16	0.10	0.11
query12	0.14	0.11	0.11
query13	0.60	0.60	0.60
query14	2.68	2.69	2.71
query15	0.92	0.85	0.85
query16	0.40	0.39	0.37
query17	1.02	1.04	1.06
query18	0.20	0.20	0.19
query19	1.88	1.91	1.86
query20	0.02	0.01	0.02
query21	15.37	0.95	0.55
query22	0.74	1.29	0.66
query23	14.85	1.41	0.62
query24	7.46	1.41	0.46
query25	0.49	0.23	0.06
query26	0.58	0.17	0.14
query27	0.05	0.05	0.05
query28	9.86	0.89	0.43
query29	12.52	3.96	3.30
query30	0.27	0.08	0.06
query31	2.83	0.58	0.40
query32	3.23	0.54	0.46
query33	2.99	3.18	3.12
query34	15.80	5.12	4.43
query35	4.52	4.51	4.46
query36	0.68	0.49	0.49
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.74 s
Total hot run time: 30.75 s

@englefly

englefly commented Apr 14, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

doris-robot commented Apr 14, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33877 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a820a50a1ab83064f2404740f955ee4d3892e0f0, data reload: false

------ Round 1 ----------------------------------
q1	26021	5103	4989	4989
q2	2071	275	176	176
q3	10399	1222	659	659
q4	10231	986	509	509
q5	7546	2334	2321	2321
q6	186	164	131	131
q7	897	737	606	606
q8	9305	1269	1032	1032
q9	6943	5079	5107	5079
q10	6814	2280	1874	1874
q11	467	291	270	270
q12	355	350	214	214
q13	17761	3673	3045	3045
q14	228	224	219	219
q15	529	488	472	472
q16	648	604	605	604
q17	570	849	348	348
q18	7557	7177	7187	7177
q19	1230	971	530	530
q20	329	321	217	217
q21	3933	2619	2398	2398
q22	1060	1025	1007	1007
Total cold run time: 115080 ms
Total hot run time: 33877 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5089	5054	5101	5054
q2	250	339	253	253
q3	2302	2734	2326	2326
q4	1430	1824	1430	1430
q5	4505	4459	4475	4459
q6	226	168	128	128
q7	2026	1956	1833	1833
q8	2619	2550	2566	2550
q9	7384	7383	7258	7258
q10	2979	3188	2747	2747
q11	576	508	493	493
q12	671	757	618	618
q13	3481	3922	3345	3345
q14	276	303	266	266
q15	511	478	474	474
q16	665	698	663	663
q17	1143	1503	1411	1411
q18	8046	7626	7415	7415
q19	871	773	829	773
q20	1922	2040	1818	1818
q21	5251	4703	4746	4703
q22	1124	1092	1061	1061
Total cold run time: 53347 ms
Total hot run time: 51078 ms

@doris-robot

doris-robot commented Apr 14, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 192778 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a820a50a1ab83064f2404740f955ee4d3892e0f0, data reload: false

query1	1410	1101	1054	1054
query2	6322	1867	1867	1867
query3	11126	4613	4591	4591
query4	25205	23810	23393	23393
query5	4060	610	458	458
query6	320	204	196	196
query7	3988	488	277	277
query8	301	251	254	251
query9	8527	2622	2589	2589
query10	467	333	255	255
query11	15368	15022	14860	14860
query12	151	111	108	108
query13	1557	520	382	382
query14	8740	5867	5878	5867
query15	198	199	172	172
query16	7569	641	499	499
query17	1197	741	586	586
query18	2039	442	330	330
query19	209	210	163	163
query20	129	133	125	125
query21	214	129	108	108
query22	4516	4670	4327	4327
query23	34515	34011	33441	33441
query24	8449	2490	2395	2395
query25	531	459	389	389
query26	1234	274	151	151
query27	2843	504	334	334
query28	4765	2435	2459	2435
query29	695	565	445	445
query30	278	233	193	193
query31	916	845	797	797
query32	73	59	64	59
query33	541	378	311	311
query34	799	883	537	537
query35	835	863	764	764
query36	982	995	888	888
query37	119	98	72	72
query38	4198	4321	4258	4258
query39	1556	1472	1443	1443
query40	218	117	134	117
query41	54	51	50	50
query42	135	110	106	106
query43	500	523	471	471
query44	1288	806	820	806
query45	182	175	175	175
query46	833	1027	644	644
query47	1875	1893	1802	1802
query48	369	421	294	294
query49	748	496	417	417
query50	649	711	399	399
query51	4322	4285	4279	4279
query52	111	112	98	98
query53	233	275	187	187
query54	594	593	524	524
query55	82	82	82	82
query56	319	318	294	294
query57	1139	1196	1126	1126
query58	266	269	264	264
query59	2732	2695	2642	2642
query60	322	333	327	327
query61	130	131	128	128
query62	771	733	684	684
query63	224	214	181	181
query64	4052	1067	697	697
query65	4415	4375	4422	4375
query66	972	412	323	323
query67	16026	15774	15334	15334
query68	9050	886	513	513
query69	475	296	262	262
query70	1192	1084	1129	1084
query71	460	328	294	294
query72	5227	4656	4684	4656
query73	710	566	344	344
query74	8865	9278	8774	8774
query75	4246	3211	2645	2645
query76	3728	1191	752	752
query77	800	376	276	276
query78	10441	10433	9392	9392
query79	1852	823	560	560
query80	621	507	438	438
query81	469	262	219	219
query82	456	123	96	96
query83	253	249	233	233
query84	249	103	88	88
query85	773	358	312	312
query86	328	325	300	300
query87	4436	4427	4449	4427
query88	3178	2185	2175	2175
query89	430	321	276	276
query90	1914	205	207	205
query91	145	141	112	112
query92	75	57	61	57
query93	1058	932	573	573
query94	654	409	308	308
query95	372	288	277	277
query96	480	553	270	270
query97	3219	3224	3156	3156
query98	229	210	199	199
query99	1587	1387	1309	1309
Total cold run time: 278475 ms
Total hot run time: 192778 ms

@doris-robot

doris-robot commented Apr 14, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.31 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a820a50a1ab83064f2404740f955ee4d3892e0f0, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.10	0.11
query3	0.25	0.20	0.19
query4	1.60	0.20	0.10
query5	0.58	0.54	0.57
query6	1.18	0.71	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.52	0.52
query10	0.57	0.57	0.58
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	2.67	2.79	2.70
query15	0.91	0.85	0.85
query16	0.39	0.39	0.38
query17	1.01	1.03	1.01
query18	0.20	0.20	0.20
query19	2.01	1.90	1.81
query20	0.01	0.01	0.01
query21	15.35	0.90	0.56
query22	0.76	1.15	0.66
query23	14.98	1.41	0.63
query24	7.07	1.76	0.95
query25	0.54	0.23	0.12
query26	0.62	0.16	0.14
query27	0.05	0.06	0.05
query28	10.35	0.86	0.41
query29	12.52	3.99	3.32
query30	0.25	0.10	0.06
query31	2.83	0.59	0.39
query32	3.24	0.55	0.46
query33	3.11	3.12	3.14
query34	15.77	5.18	4.52
query35	4.53	4.55	4.56
query36	0.65	0.51	0.47
query37	0.08	0.06	0.07
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.19 s
Total hot run time: 31.31 s

@englefly

englefly commented Apr 14, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

doris-robot commented Apr 14, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 34954 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 619bfc0cacd6f380e35f39fd6c614b71a7057493, data reload: false

------ Round 1 ----------------------------------
q1	25864	5111	5037	5037
q2	2073	277	186	186
q3	11070	1266	712	712
q4	10247	991	539	539
q5	8792	2440	2313	2313
q6	187	163	134	134
q7	901	742	616	616
q8	9312	1277	1047	1047
q9	6845	5084	5109	5084
q10	6877	2306	1904	1904
q11	485	285	279	279
q12	359	351	233	233
q13	17887	3790	3172	3172
q14	219	245	208	208
q15	543	485	490	485
q16	632	612	593	593
q17	569	849	352	352
q18	9118	7759	7694	7694
q19	4134	1007	537	537
q20	359	321	216	216
q21	3897	2615	3115	2615
q22	1077	1046	998	998
Total cold run time: 121447 ms
Total hot run time: 34954 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5306	5092	5130	5092
q2	253	325	232	232
q3	2086	2755	2451	2451
q4	1509	1877	1483	1483
q5	4450	4367	4333	4333
q6	217	168	126	126
q7	1953	1896	1729	1729
q8	2616	2509	2489	2489
q9	7024	7112	7342	7112
q10	2983	3231	2788	2788
q11	587	522	493	493
q12	709	802	614	614
q13	3667	4075	3370	3370
q14	279	300	264	264
q15	544	476	482	476
q16	641	685	650	650
q17	1123	1520	1384	1384
q18	7621	7456	7230	7230
q19	802	823	893	823
q20	1949	1989	1853	1853
q21	5299	4932	4775	4775
q22	1146	1096	1012	1012
Total cold run time: 52764 ms
Total hot run time: 50779 ms

@doris-robot

doris-robot commented Apr 14, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 193402 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 619bfc0cacd6f380e35f39fd6c614b71a7057493, data reload: false

query1	1411	1094	1015	1015
query2	6272	1839	1846	1839
query3	11129	4559	4559	4559
query4	57824	24284	23556	23556
query5	4836	531	515	515
query6	374	214	205	205
query7	4926	515	277	277
query8	353	261	252	252
query9	6644	2584	2625	2584
query10	420	345	270	270
query11	15066	15168	15025	15025
query12	157	112	105	105
query13	1073	516	387	387
query14	10104	6088	6154	6088
query15	209	196	179	179
query16	7196	660	502	502
query17	1009	734	569	569
query18	1530	399	323	323
query19	186	177	157	157
query20	123	123	124	123
query21	203	125	105	105
query22	4297	4455	4401	4401
query23	33816	33485	33744	33485
query24	6788	2439	2454	2439
query25	459	481	391	391
query26	662	267	150	150
query27	2262	510	331	331
query28	3328	2444	2438	2438
query29	585	565	451	451
query30	272	222	196	196
query31	882	880	791	791
query32	76	67	61	61
query33	443	372	304	304
query34	773	876	514	514
query35	790	839	767	767
query36	981	1003	918	918
query37	120	97	78	78
query38	4321	4322	4203	4203
query39	1487	1425	1427	1425
query40	205	126	119	119
query41	63	66	62	62
query42	135	120	117	117
query43	528	523	481	481
query44	1292	845	833	833
query45	184	185	176	176
query46	849	1022	641	641
query47	1846	1890	1762	1762
query48	388	414	303	303
query49	724	544	426	426
query50	668	694	441	441
query51	4291	4402	4278	4278
query52	112	109	107	107
query53	231	261	190	190
query54	587	575	527	527
query55	84	88	91	88
query56	307	327	296	296
query57	1170	1128	1064	1064
query58	259	250	265	250
query59	2553	2635	2458	2458
query60	326	313	315	313
query61	130	125	128	125
query62	705	711	654	654
query63	232	183	181	181
query64	1447	1025	683	683
query65	4310	4197	4228	4197
query66	704	394	299	299
query67	15726	15522	15431	15431
query68	6527	817	518	518
query69	557	288	268	268
query70	1187	1126	1098	1098
query71	529	308	302	302
query72	5981	4815	4975	4815
query73	1302	693	344	344
query74	9273	9113	9021	9021
query75	3888	3185	2696	2696
query76	4431	1196	825	825
query77	625	362	280	280
query78	10094	10268	9272	9272
query79	2107	824	574	574
query80	610	515	445	445
query81	497	265	220	220
query82	456	123	94	94
query83	249	258	230	230
query84	296	105	81	81
query85	804	402	315	315
query86	382	285	278	278
query87	4494	4406	4321	4321
query88	3456	2231	2243	2231
query89	403	314	276	276
query90	1891	205	203	203
query91	142	141	108	108
query92	74	58	60	58
query93	1742	942	581	581
query94	660	411	306	306
query95	370	286	286	286
query96	483	572	278	278
query97	3232	3223	3108	3108
query98	227	210	203	203
query99	1406	1413	1287	1287
Total cold run time: 301818 ms
Total hot run time: 193402 ms

@doris-robot

doris-robot commented Apr 14, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.47 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 619bfc0cacd6f380e35f39fd6c614b71a7057493, data reload: false

query1	0.04	0.03	0.03
query2	0.13	0.10	0.11
query3	0.25	0.20	0.20
query4	1.60	0.19	0.20
query5	0.60	0.58	0.60
query6	1.19	0.71	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.52	0.51
query10	0.58	0.59	0.57
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.69	2.69	2.78
query15	0.92	0.85	0.86
query16	0.38	0.38	0.38
query17	1.02	1.02	1.03
query18	0.21	0.20	0.20
query19	1.86	1.92	1.79
query20	0.02	0.01	0.01
query21	15.35	0.92	0.57
query22	0.76	1.22	0.67
query23	14.91	1.37	0.61
query24	6.85	1.22	1.14
query25	0.48	0.27	0.06
query26	0.60	0.16	0.15
query27	0.06	0.06	0.05
query28	9.72	0.86	0.44
query29	12.54	4.05	3.29
query30	0.25	0.09	0.06
query31	2.84	0.59	0.39
query32	3.23	0.54	0.48
query33	2.99	3.09	3.04
query34	15.76	5.12	4.51
query35	4.51	4.57	4.54
query36	0.67	0.50	0.48
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.05	0.03
Total cold run time: 105.04 s
Total hot run time: 31.47 s

@englefly englefly force-pushed the set-runtime-filter-fe-0410 branch from 619bfc0 to 9329f0c Compare April 14, 2025 09:39
@englefly

englefly commented Apr 14, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

doris-robot commented Apr 14, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33838 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9329f0c7a3555604f8f92c39aef7c5f89ab1d431, data reload: false

------ Round 1 ----------------------------------
q1	25786	5497	5017	5017
q2	2074	276	183	183
q3	10387	1200	699	699
q4	10231	980	517	517
q5	7540	2349	2240	2240
q6	181	160	132	132
q7	896	738	624	624
q8	9308	1171	1090	1090
q9	6743	5136	5088	5088
q10	6806	2284	1863	1863
q11	469	284	266	266
q12	349	346	222	222
q13	17775	3614	3098	3098
q14	219	230	212	212
q15	529	486	500	486
q16	615	613	580	580
q17	579	846	335	335
q18	7393	7165	7110	7110
q19	1475	937	530	530
q20	327	320	217	217
q21	3729	3293	2372	2372
q22	1025	1027	957	957
Total cold run time: 114436 ms
Total hot run time: 33838 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5111	5041	5005	5005
q2	237	323	236	236
q3	2119	2624	2283	2283
q4	1386	1758	1361	1361
q5	4396	4361	4380	4361
q6	211	166	128	128
q7	1948	1930	1753	1753
q8	2560	2520	2627	2520
q9	7413	7244	6995	6995
q10	3063	3233	2830	2830
q11	588	531	492	492
q12	706	770	642	642
q13	3568	4039	3407	3407
q14	276	316	305	305
q15	549	509	497	497
q16	643	685	656	656
q17	1178	1562	1401	1401
q18	7849	7584	7417	7417
q19	801	820	912	820
q20	1945	2036	1851	1851
q21	5194	4662	4830	4662
q22	1107	1056	1002	1002
Total cold run time: 52848 ms
Total hot run time: 50624 ms

@doris-robot

doris-robot commented Apr 14, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 192434 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9329f0c7a3555604f8f92c39aef7c5f89ab1d431, data reload: false

query1	1423	1102	1042	1042
query2	6210	1881	1880	1880
query3	10962	4653	4421	4421
query4	56169	25464	22917	22917
query5	5140	481	473	473
query6	368	203	187	187
query7	4979	500	281	281
query8	316	254	244	244
query9	6057	2581	2583	2581
query10	401	334	278	278
query11	15072	14988	14860	14860
query12	160	116	110	110
query13	1108	508	383	383
query14	10039	6151	6071	6071
query15	207	202	177	177
query16	7194	667	475	475
query17	1080	764	571	571
query18	1576	397	337	337
query19	188	187	170	170
query20	126	114	124	114
query21	212	122	110	110
query22	4258	4292	4341	4292
query23	34007	33558	33520	33520
query24	6521	2459	2436	2436
query25	503	478	413	413
query26	678	277	149	149
query27	2223	510	337	337
query28	3299	2416	2423	2416
query29	574	546	438	438
query30	282	222	195	195
query31	873	858	794	794
query32	79	64	64	64
query33	480	351	311	311
query34	769	869	510	510
query35	818	840	776	776
query36	950	1001	925	925
query37	125	112	84	84
query38	4254	4213	4227	4213
query39	1500	1461	1434	1434
query40	239	130	114	114
query41	60	57	55	55
query42	122	116	111	111
query43	511	508	491	491
query44	1348	852	852	852
query45	180	178	169	169
query46	851	1033	645	645
query47	1802	1863	1797	1797
query48	398	443	311	311
query49	711	565	460	460
query50	677	686	424	424
query51	4241	4235	4195	4195
query52	109	112	100	100
query53	256	262	200	200
query54	586	575	529	529
query55	90	82	81	81
query56	306	300	291	291
query57	1151	1187	1117	1117
query58	261	251	296	251
query59	2593	2795	2703	2703
query60	334	320	308	308
query61	137	130	131	130
query62	737	744	678	678
query63	229	186	190	186
query64	1503	1041	682	682
query65	4320	4251	4229	4229
query66	708	409	306	306
query67	15718	15513	15283	15283
query68	8132	875	518	518
query69	534	295	261	261
query70	1182	1105	1061	1061
query71	526	319	289	289
query72	5719	4843	4860	4843
query73	1365	667	339	339
query74	9189	9172	9000	9000
query75	3698	3183	2667	2667
query76	4364	1188	763	763
query77	618	363	288	288
query78	9860	9992	9277	9277
query79	6418	795	551	551
query80	686	516	437	437
query81	483	256	221	221
query82	733	127	95	95
query83	347	254	234	234
query84	287	105	86	86
query85	810	371	317	317
query86	397	321	275	275
query87	4569	4529	4370	4370
query88	3520	2214	2204	2204
query89	451	317	272	272
query90	1930	206	206	206
query91	169	143	114	114
query92	80	60	58	58
query93	3075	925	577	577
query94	690	421	304	304
query95	368	299	278	278
query96	482	564	273	273
query97	3176	3194	3111	3111
query98	235	200	201	200
query99	1432	1413	1258	1258
Total cold run time: 306883 ms
Total hot run time: 192434 ms

@doris-robot

doris-robot commented Apr 14, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.6 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9329f0c7a3555604f8f92c39aef7c5f89ab1d431, data reload: false

query1	0.03	0.04	0.03
query2	0.13	0.11	0.10
query3	0.25	0.20	0.20
query4	1.60	0.20	0.19
query5	0.58	0.56	0.55
query6	1.18	0.72	0.72
query7	0.03	0.01	0.02
query8	0.04	0.03	0.03
query9	0.57	0.53	0.52
query10	0.58	0.59	0.58
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.59	0.59
query14	2.79	2.69	2.71
query15	0.92	0.86	0.87
query16	0.39	0.38	0.38
query17	1.03	1.00	1.06
query18	0.21	0.19	0.20
query19	2.00	1.85	1.81
query20	0.01	0.01	0.01
query21	15.36	0.92	0.57
query22	0.75	1.14	0.67
query23	15.02	1.39	0.65
query24	6.82	1.32	1.59
query25	0.47	0.22	0.06
query26	0.63	0.18	0.12
query27	0.05	0.06	0.05
query28	9.51	0.87	0.43
query29	12.54	4.06	3.33
query30	0.24	0.09	0.07
query31	2.83	0.60	0.38
query32	3.23	0.54	0.47
query33	3.01	3.11	3.04
query34	15.61	5.07	4.45
query35	4.58	4.51	4.50
query36	0.68	0.49	0.50
query37	0.09	0.06	0.06
query38	0.06	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.09 s
Total hot run time: 31.6 s

@englefly

englefly commented Apr 14, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

doris-robot commented Apr 14, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33888 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 14756201ffb8c9960a242988523af84ee3c25f21, data reload: false

------ Round 1 ----------------------------------
q1	25893	5075	5050	5050
q2	2055	274	183	183
q3	10397	1251	689	689
q4	10232	988	516	516
q5	7539	2320	2287	2287
q6	187	162	134	134
q7	906	736	596	596
q8	9301	1211	1019	1019
q9	6726	5067	5135	5067
q10	6804	2297	1868	1868
q11	457	290	275	275
q12	346	345	214	214
q13	17759	3639	3094	3094
q14	229	232	204	204
q15	539	490	491	490
q16	641	619	584	584
q17	576	840	348	348
q18	7599	7087	7097	7087
q19	1236	933	549	549
q20	325	317	220	220
q21	4246	2595	2443	2443
q22	1059	1011	971	971
Total cold run time: 115052 ms
Total hot run time: 33888 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5027	5118	5028	5028
q2	247	322	230	230
q3	2162	2611	2323	2323
q4	1410	1815	1384	1384
q5	4362	4377	4452	4377
q6	209	167	128	128
q7	2026	1939	1761	1761
q8	2573	2535	2505	2505
q9	7237	7216	7089	7089
q10	2964	3202	2722	2722
q11	584	498	505	498
q12	672	786	612	612
q13	3472	3874	3290	3290
q14	290	308	277	277
q15	539	482	476	476
q16	631	691	643	643
q17	1147	1515	1406	1406
q18	7730	7530	7403	7403
q19	790	785	792	785
q20	1919	1980	1870	1870
q21	5271	4856	4802	4802
q22	1077	1042	996	996
Total cold run time: 52339 ms
Total hot run time: 50605 ms

@doris-robot

doris-robot commented Apr 14, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 192428 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 14756201ffb8c9960a242988523af84ee3c25f21, data reload: false

query1	1401	1148	1096	1096
query2	6159	1865	1871	1865
query3	11055	4558	4365	4365
query4	52262	25078	22925	22925
query5	5103	557	453	453
query6	344	219	207	207
query7	4887	484	280	280
query8	312	268	257	257
query9	5416	2623	2633	2623
query10	442	312	269	269
query11	15007	15125	14790	14790
query12	154	113	104	104
query13	1066	522	401	401
query14	9950	6123	6094	6094
query15	208	207	182	182
query16	7132	671	478	478
query17	1086	764	607	607
query18	1565	410	330	330
query19	199	196	173	173
query20	140	124	132	124
query21	207	127	113	113
query22	4349	4535	4265	4265
query23	34257	33559	33540	33540
query24	6961	2425	2456	2425
query25	472	462	414	414
query26	715	307	158	158
query27	2522	493	336	336
query28	3268	2441	2439	2439
query29	579	554	429	429
query30	267	223	200	200
query31	861	870	775	775
query32	71	67	64	64
query33	481	370	316	316
query34	788	867	531	531
query35	814	837	752	752
query36	949	1021	891	891
query37	119	100	77	77
query38	4174	4295	4170	4170
query39	1507	1461	1467	1461
query40	232	120	112	112
query41	60	52	50	50
query42	122	106	108	106
query43	512	507	472	472
query44	1313	825	832	825
query45	180	180	172	172
query46	849	1033	643	643
query47	1874	1911	1807	1807
query48	387	405	300	300
query49	694	506	447	447
query50	685	694	418	418
query51	4216	4337	4215	4215
query52	107	114	103	103
query53	227	258	179	179
query54	576	591	524	524
query55	92	94	87	87
query56	337	300	285	285
query57	1162	1159	1108	1108
query58	270	264	266	264
query59	2638	2810	2663	2663
query60	345	314	302	302
query61	135	130	123	123
query62	741	741	666	666
query63	224	185	187	185
query64	2158	1043	706	706
query65	4272	4244	4225	4225
query66	744	394	300	300
query67	15863	15618	15485	15485
query68	7489	810	519	519
query69	525	305	262	262
query70	1216	1154	1109	1109
query71	493	303	292	292
query72	6012	4771	4801	4771
query73	1491	636	346	346
query74	9183	8919	8878	8878
query75	4007	3189	2690	2690
query76	4250	1196	742	742
query77	690	362	349	349
query78	10014	10158	9189	9189
query79	2587	803	569	569
query80	626	503	433	433
query81	478	254	226	226
query82	469	126	99	99
query83	330	249	232	232
query84	297	101	88	88
query85	783	350	311	311
query86	362	320	276	276
query87	4440	4453	4350	4350
query88	3737	2229	2222	2222
query89	407	317	280	280
query90	1850	212	207	207
query91	140	139	112	112
query92	77	62	58	58
query93	1917	934	571	571
query94	663	415	308	308
query95	365	295	294	294
query96	484	555	273	273
query97	3178	3226	3127	3127
query98	240	196	198	196
query99	1449	1401	1298	1298
Total cold run time: 298513 ms
Total hot run time: 192428 ms

@doris-robot

doris-robot commented Apr 14, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.71 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 14756201ffb8c9960a242988523af84ee3c25f21, data reload: false

query1	0.03	0.03	0.03
query2	0.12	0.10	0.11
query3	0.26	0.20	0.19
query4	1.60	0.19	0.19
query5	0.59	0.59	0.57
query6	1.19	0.72	0.72
query7	0.02	0.01	0.01
query8	0.03	0.04	0.04
query9	0.57	0.54	0.51
query10	0.57	0.58	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.71	2.84	2.79
query15	0.90	0.85	0.86
query16	0.38	0.37	0.39
query17	1.01	1.01	1.00
query18	0.21	0.19	0.20
query19	1.88	1.96	1.80
query20	0.02	0.01	0.01
query21	15.35	0.88	0.56
query22	0.76	1.06	0.62
query23	15.06	1.37	0.62
query24	7.39	0.82	0.44
query25	0.50	0.13	0.09
query26	0.66	0.17	0.14
query27	0.05	0.05	0.04
query28	9.29	0.86	0.43
query29	12.53	3.95	3.29
query30	0.26	0.08	0.06
query31	2.82	0.59	0.40
query32	3.24	0.54	0.46
query33	3.05	3.02	3.06
query34	15.83	5.07	4.52
query35	4.52	4.47	4.52
query36	0.66	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.18	0.13	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.46 s
Total hot run time: 30.71 s

@englefly

englefly commented Apr 15, 2025

Copy link
Copy Markdown
Contributor Author

run cloud_p0

@englefly

englefly commented Apr 15, 2025

Copy link
Copy Markdown
Contributor Author

run p0

@BiteTheDDDDt

BiteTheDDDDt commented Apr 15, 2025

Copy link
Copy Markdown
Contributor

run buildall

@doris-robot

doris-robot commented Apr 15, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 34214 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 79cf7202fbfdd9dd48e66bb1139d29e64084f2e1, data reload: false

------ Round 1 ----------------------------------
q1	26154	5178	5113	5113
q2	2061	270	184	184
q3	10411	1262	699	699
q4	10228	1016	525	525
q5	7553	2415	2338	2338
q6	195	166	135	135
q7	921	745	596	596
q8	9315	1293	1118	1118
q9	6779	5067	5134	5067
q10	6834	2335	1934	1934
q11	470	285	266	266
q12	362	371	218	218
q13	17756	3726	3124	3124
q14	230	233	217	217
q15	549	497	490	490
q16	625	621	584	584
q17	572	865	355	355
q18	7606	7175	7036	7036
q19	1085	943	564	564
q20	344	324	217	217
q21	3959	3325	2418	2418
q22	1064	1026	1016	1016
Total cold run time: 115073 ms
Total hot run time: 34214 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5147	5117	5102	5102
q2	247	335	229	229
q3	2169	2673	2293	2293
q4	1455	1884	1450	1450
q5	4484	4412	4399	4399
q6	217	168	130	130
q7	1966	1973	1823	1823
q8	2585	2520	2595	2520
q9	7246	7103	7094	7094
q10	3009	3200	2749	2749
q11	570	511	500	500
q12	711	758	631	631
q13	3553	3974	3314	3314
q14	294	296	281	281
q15	527	496	470	470
q16	652	691	644	644
q17	1171	1526	1454	1454
q18	7742	7460	7261	7261
q19	836	879	915	879
q20	1957	2061	1862	1862
q21	5471	4987	4796	4796
q22	1168	1075	1036	1036
Total cold run time: 53177 ms
Total hot run time: 50917 ms

@doris-robot

doris-robot commented Jun 24, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 185512 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 81ea852500cea5914900ea414bfbb298a2dd30ab, data reload: false

query1	974	396	388	388
query2	2588	1890	1849	1849
query3	6494	221	219	219
query4	26207	23353	22939	22939
query5	2311	645	496	496
query6	311	226	208	208
query7	4581	491	289	289
query8	292	234	227	227
query9	8213	2612	2639	2612
query10	476	338	265	265
query11	15816	15068	14860	14860
query12	161	108	104	104
query13	1635	545	413	413
query14	8845	5937	5981	5937
query15	210	199	178	178
query16	6027	617	482	482
query17	1331	732	552	552
query18	2021	398	300	300
query19	194	196	153	153
query20	122	116	115	115
query21	191	121	109	109
query22	4063	4105	4059	4059
query23	33962	33141	33149	33141
query24	8437	2341	2378	2341
query25	477	466	394	394
query26	1067	270	161	161
query27	2773	518	344	344
query28	4328	2163	2132	2132
query29	780	544	426	426
query30	335	220	198	198
query31	903	818	754	754
query32	73	68	66	66
query33	538	362	322	322
query34	814	856	550	550
query35	823	876	760	760
query36	944	970	908	908
query37	118	108	77	77
query38	4149	4131	4018	4018
query39	1510	1418	1414	1414
query40	217	117	110	110
query41	65	59	71	59
query42	130	111	112	111
query43	497	511	462	462
query44	1382	837	842	837
query45	176	172	182	172
query46	871	1024	634	634
query47	1739	1803	1703	1703
query48	409	425	329	329
query49	813	492	407	407
query50	658	690	403	403
query51	4130	4135	4077	4077
query52	113	109	110	109
query53	233	252	188	188
query54	584	578	516	516
query55	89	83	86	83
query56	311	308	286	286
query57	1183	1203	1152	1152
query58	294	286	267	267
query59	2525	2654	2560	2560
query60	346	335	325	325
query61	158	145	142	142
query62	832	725	734	725
query63	229	192	199	192
query64	3598	1110	776	776
query65	4266	4165	4210	4165
query66	1137	435	335	335
query67	15946	15470	15392	15392
query68	7954	909	553	553
query69	497	319	334	319
query70	1160	1116	1077	1077
query71	483	340	291	291
query72	5527	4736	4841	4736
query73	691	614	350	350
query74	8865	9074	8611	8611
query75	3694	3187	2693	2693
query76	3556	1203	752	752
query77	741	395	287	287
query78	10017	10200	9289	9289
query79	2599	824	625	625
query80	669	531	482	482
query81	494	259	224	224
query82	461	141	99	99
query83	267	263	244	244
query84	248	111	88	88
query85	788	365	314	314
query86	389	308	286	286
query87	4446	4421	4354	4354
query88	4053	2301	2289	2289
query89	389	315	288	288
query90	1868	228	213	213
query91	142	144	116	116
query92	76	65	58	58
query93	2306	985	602	602
query94	653	402	310	310
query95	376	293	290	290
query96	526	574	281	281
query97	2737	2747	2618	2618
query98	229	217	207	207
query99	1414	1402	1289	1289
Total cold run time: 266340 ms
Total hot run time: 185512 ms

@doris-robot

doris-robot commented Jun 24, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.43 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 81ea852500cea5914900ea414bfbb298a2dd30ab, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.04	0.04
query3	0.24	0.07	0.06
query4	1.60	0.10	0.11
query5	0.43	0.39	0.41
query6	1.17	0.65	0.67
query7	0.03	0.01	0.02
query8	0.04	0.04	0.03
query9	0.60	0.51	0.50
query10	0.56	0.57	0.56
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.63	0.61	0.60
query14	0.81	0.82	0.82
query15	0.91	0.87	0.86
query16	0.39	0.38	0.39
query17	1.09	1.06	1.08
query18	0.24	0.21	0.21
query19	1.99	1.80	1.80
query20	0.01	0.01	0.01
query21	15.42	0.90	0.53
query22	0.76	1.09	0.59
query23	15.08	1.37	0.62
query24	6.95	1.67	0.90
query25	0.52	0.25	0.07
query26	0.48	0.16	0.14
query27	0.07	0.06	0.06
query28	10.03	0.89	0.44
query29	12.53	3.96	3.33
query30	0.26	0.09	0.06
query31	2.84	0.60	0.39
query32	3.24	0.55	0.47
query33	3.10	3.10	3.18
query34	16.06	5.40	4.75
query35	4.87	4.83	4.82
query36	0.68	0.53	0.51
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.18	0.15	0.14
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 104.54 s
Total hot run time: 29.43 s

morrySnow pushed a commit that referenced this pull request Jun 24, 2025
@englefly
branch-3.1: [fix](nereids) fix bug for Logical/PhysicalIntersect.cons…
bdf4a32 
…tructReplaceMap #49947 (#52169)

Cherry-pick part of #49947
@englefly

englefly commented Jun 24, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@englefly

englefly commented Jun 25, 2025

Copy link
Copy Markdown
Contributor Author

run performance

@doris-robot

doris-robot commented Jun 25, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33884 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c460f6563a7a59fdcb8f13dc674738d81d2d233e, data reload: false

------ Round 1 ----------------------------------
q1	17611	5242	5021	5021
q2	1962	301	189	189
q3	10443	1307	726	726
q4	10242	999	523	523
q5	7660	2359	2346	2346
q6	182	165	131	131
q7	895	737	620	620
q8	9312	1301	1095	1095
q9	6902	5045	5056	5045
q10	6908	2385	1985	1985
q11	486	281	274	274
q12	344	347	222	222
q13	17758	3686	3068	3068
q14	254	226	215	215
q15	565	481	494	481
q16	424	423	369	369
q17	586	879	351	351
q18	7488	7277	7124	7124
q19	1728	949	550	550
q20	332	343	221	221
q21	3653	2617	2370	2370
q22	1034	1016	958	958
Total cold run time: 106769 ms
Total hot run time: 33884 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5159	5056	5452	5056
q2	243	316	228	228
q3	2162	2645	2312	2312
q4	1346	1765	1349	1349
q5	4241	4422	4359	4359
q6	210	169	129	129
q7	2028	1988	1781	1781
q8	2573	2800	2479	2479
q9	7151	7037	7107	7037
q10	3102	3295	2831	2831
q11	593	517	493	493
q12	695	782	624	624
q13	3496	3960	3369	3369
q14	294	292	270	270
q15	541	481	472	472
q16	429	475	421	421
q17	1097	1433	1336	1336
q18	7396	7176	7120	7120
q19	762	755	850	755
q20	1930	1980	1817	1817
q21	4826	4456	4439	4439
q22	1043	1043	1006	1006
Total cold run time: 51317 ms
Total hot run time: 49683 ms

@doris-robot

doris-robot commented Jun 25, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 186002 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c460f6563a7a59fdcb8f13dc674738d81d2d233e, data reload: false

query1	978	381	401	381
query2	6541	1865	1819	1819
query3	6740	216	213	213
query4	26685	23540	23520	23520
query5	4336	651	459	459
query6	313	216	210	210
query7	4640	503	307	307
query8	273	245	221	221
query9	8642	2643	2663	2643
query10	478	330	274	274
query11	15862	15068	14886	14886
query12	173	117	107	107
query13	1662	541	427	427
query14	9345	6016	5950	5950
query15	214	200	175	175
query16	7381	626	472	472
query17	1226	717	601	601
query18	2001	409	314	314
query19	200	242	150	150
query20	121	117	109	109
query21	210	134	113	113
query22	4128	4233	4111	4111
query23	33844	32993	33094	32993
query24	8494	2353	2412	2353
query25	536	476	392	392
query26	1084	271	153	153
query27	2733	511	342	342
query28	4287	2137	2111	2111
query29	762	544	437	437
query30	285	214	189	189
query31	940	843	738	738
query32	72	68	63	63
query33	556	404	312	312
query34	784	867	528	528
query35	801	837	752	752
query36	957	958	868	868
query37	119	99	73	73
query38	4104	4097	4084	4084
query39	1464	1439	1410	1410
query40	215	118	105	105
query41	73	71	72	71
query42	135	117	114	114
query43	503	529	489	489
query44	1308	820	806	806
query45	185	178	166	166
query46	857	1023	631	631
query47	1758	1794	1695	1695
query48	400	426	330	330
query49	749	491	391	391
query50	634	673	404	404
query51	4139	4138	4102	4102
query52	113	118	105	105
query53	223	252	192	192
query54	583	570	511	511
query55	86	87	81	81
query56	314	324	305	305
query57	1168	1208	1109	1109
query58	270	262	252	252
query59	2647	2722	2637	2637
query60	333	319	310	310
query61	125	131	130	130
query62	801	705	683	683
query63	227	204	194	194
query64	4015	1167	734	734
query65	4262	4156	4209	4156
query66	1085	413	343	343
query67	15645	15650	15492	15492
query68	9293	887	533	533
query69	475	304	272	272
query70	1144	1121	1104	1104
query71	463	320	313	313
query72	5562	4702	4749	4702
query73	704	605	352	352
query74	9099	8773	8750	8750
query75	4293	3166	2720	2720
query76	3701	1192	756	756
query77	998	377	293	293
query78	10015	10246	9231	9231
query79	1420	877	601	601
query80	595	523	462	462
query81	477	257	234	234
query82	431	125	97	97
query83	246	250	238	238
query84	250	107	82	82
query85	795	420	327	327
query86	341	295	292	292
query87	4383	4464	4350	4350
query88	2865	2279	2315	2279
query89	383	317	291	291
query90	1924	214	216	214
query91	145	146	116	116
query92	75	63	64	63
query93	1052	943	581	581
query94	673	399	306	306
query95	370	283	287	283
query96	489	567	283	283
query97	2702	2705	2614	2614
query98	238	215	206	206
query99	1602	1395	1259	1259
Total cold run time: 274027 ms
Total hot run time: 186002 ms

@doris-robot

doris-robot commented Jun 25, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.09 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c460f6563a7a59fdcb8f13dc674738d81d2d233e, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.42	0.41	0.42
query6	1.17	0.66	0.67
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.56	0.51	0.52
query10	0.56	0.58	0.58
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.63	0.62	0.61
query14	0.79	0.82	0.82
query15	0.91	0.87	0.88
query16	0.38	0.39	0.39
query17	1.08	1.07	1.09
query18	0.23	0.22	0.22
query19	1.94	1.88	1.81
query20	0.02	0.01	0.02
query21	15.38	0.90	0.55
query22	0.75	1.10	0.63
query23	15.05	1.38	0.65
query24	7.58	0.70	0.39
query25	0.46	0.26	0.08
query26	0.64	0.16	0.14
query27	0.06	0.05	0.06
query28	8.80	0.95	0.44
query29	12.57	3.95	3.29
query30	0.25	0.09	0.07
query31	2.84	0.61	0.40
query32	3.23	0.55	0.47
query33	3.07	3.06	3.10
query34	16.11	5.36	4.82
query35	4.89	4.88	4.80
query36	0.70	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.02	0.02	0.02
query40	0.17	0.15	0.15
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.04
Total cold run time: 103.85 s
Total hot run time: 29.09 s

@github-actions

github-actions Bot commented Jun 25, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions Bot added approved Indicates a PR has been approved by one committer. reviewed labels Jun 25, 2025
@github-actions

github-actions Bot commented Jun 25, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@englefly englefly changed the title set-filter-fe-part [feat](nereids) support runtime filter for except/intersect (FE part) Jun 25, 2025
@englefly englefly merged commit deb2523 into apache:master Jun 25, 2025
26 of 28 checks passed
englefly added a commit that referenced this pull request Jun 30, 2025
@englefly
[fix](nereids) intersect/except runtime filter project wrong child sl…
ccc336c 
…ot (#52435)

### What problem does this PR solve?
a bug introduced by #49947 
tpcds 38/87 are involved
koarz pushed a commit to koarz/doris that referenced this pull request Jul 4, 2025
@englefly
[fix](nereids) intersect/except runtime filter project wrong child sl…
18dd6f9 
…ot (apache#52435)

### What problem does this PR solve?
a bug introduced by apache#49947 
tpcds 38/87 are involved
koarz pushed a commit to koarz/doris that referenced this pull request Jul 4, 2025
@englefly
[fix](nereids) intersect/except runtime filter project wrong child sl…
119ff85 
…ot (apache#52435)

### What problem does this PR solve?
a bug introduced by apache#49947 
tpcds 38/87 are involved
koarz pushed a commit to koarz/doris that referenced this pull request Jul 4, 2025
@englefly
[fix](nereids) intersect/except runtime filter project wrong child sl…
784a6f3 
…ot (apache#52435)

### What problem does this PR solve?
a bug introduced by apache#49947 
tpcds 38/87 are involved
dataroaring pushed a commit that referenced this pull request Jul 7, 2025
@englefly
branch-3.0 [fix](nereids) fix bug in LogicalIntersect datatrait Outof…
3a6a2b6 
…Bounds exception (#52170)

pick part of #49947
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes
@starocean999 starocean999 starocean999 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. not-merge/3.0 not-merge/3.1 reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@englefly @hello-stephen @doris-robot @BiteTheDDDDt @aytrack @starocean999 @dataroaring @morrySnow